An Embedding of ReLU Networks and an Analysis of Their Identifiability

Neural networks with the rectified linear unit (ReLU) nonlinearity are described by a vector of parameters $$\theta $$ and realized as piecewise continuous function $$\varvec{R}_{\theta }: x \in \mathbb {R}^{d} \mapsto \varvec{R}_{\theta }(x) {R}^{k}$$ . Natural scalings permutations operations on leave realization unchanged, leading to equivalence classes that yield same realization. These considerations in turn lead notion identifiability—the ability recover (the class of) from sole knowledge its }$$ The overall objective this paper is introduce an embedding for ReLU neural any depth, $${\varvec{\Phi }}(\theta )$$ , invariant provides locally parameterization network. Leveraging these two key properties, we derive some conditions under which deep network indeed identifiable finite set samples $$x_{i} {R}^{d}$$ We study shallow case more establishing necessary sufficient be bounded subset $$\mathcal {X}\subseteq